fix: multi-stage Dockerfile with build/dev/prod targets and ARM64 support#3335
Open
deacon-mp wants to merge 11 commits into
Open
fix: multi-stage Dockerfile with build/dev/prod targets and ARM64 support#3335deacon-mp wants to merge 11 commits into
deacon-mp wants to merge 11 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Reworks containerization to support explicit multi-stage Docker builds (build/dev/prod), multi-arch installs (amd64/arm64), and a slimmer dev workflow via compose.
Changes:
- Replaces single-stage runtime build with
build,dev, andprodDocker stages and copies built artifacts/venv forward. - Adds TARGETARCH-aware Go/Node installation via direct downloads; updates compose to target
devwithslimby default. - Adjusts build context ignores to allow submodule initialization inputs (e.g.,
.gitmodules).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| Dockerfile | Introduces build/dev/prod stages, multi-arch Go/Node installs, venv propagation, and non-root prod user. |
| docker-compose.yml | Targets dev stage, defaults build arg VARIANT to slim, and updates command invocation. |
| .dockerignore | Stops excluding .gitmodules (and now also no longer excludes .git). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+25
to
+54
| # Install Node | ||
| ARG TARGETARCH | ||
| ARG NODE_VERSION | ||
| RUN set -eux; \ | ||
| arch="${TARGETARCH:-amd64}"; \ | ||
| case "$arch" in \ | ||
| amd64) node_arch="x64" ;; \ | ||
| arm64) node_arch="arm64" ;; \ | ||
| *) node_arch="x64" ;; \ | ||
| esac; \ | ||
| curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${node_arch}.tar.xz" -o /tmp/node.tar.xz; \ | ||
| mkdir -p /usr/local/lib/node; \ | ||
| tar -xJf /tmp/node.tar.xz -C /usr/local/lib/node --strip-components=1; \ | ||
| rm -f /tmp/node.tar.xz | ||
|
|
||
| # Install Go | ||
| ARG GO_VERSION | ||
| RUN set -eux; \ | ||
| arch="${TARGETARCH:-amd64}"; \ | ||
| case "$arch" in \ | ||
| amd64|arm64) \ | ||
| go_arch="$arch"; \ | ||
| echo "Installing Go ${GO_VERSION} for ${go_arch}"; \ | ||
| curl -fsSL "https://go.dev/dl/go${GO_VERSION}.linux-${go_arch}.tar.gz" -o /tmp/go.tgz; \ | ||
| tar -C /usr/local -xzf /tmp/go.tgz; \ | ||
| rm -f /tmp/go.tgz ;; \ | ||
| *) \ | ||
| echo "Unsupported arch ${arch}, ignoring Go install"; \ | ||
| mkdir -p /usr/local/go/bin && touch /usr/local/go/bin/.install_failed ;; \ | ||
| esac |
Comment on lines
+103
to
+104
| RUN (find ${APP_DIR} -type d -name ".git") | xargs rm -rf \ | ||
| && rm ${APP_DIR}/.gitmodules |
Comment on lines
+185
to
+188
| # Build VueJS front-end | ||
| RUN cd ${APP_DIR}/plugins/magma \ | ||
| && npm install \ | ||
| && npm run build |
| && npm install \ | ||
| && npm run build | ||
|
|
||
| CMD ["python3", "/usr/src/app/server.py", "--insecure"] |
| docker-compose.yml | ||
| Dockerfile | ||
|
|
||
| # git |
- Add SHA256 checksum verification for Node and Go downloads to guard against supply-chain / MITM issues - Fix .git removal step: use xargs -r to handle empty find output and rm -f for .gitmodules to make the step idempotent - Move VueJS UI build from prod stage to build stage so prod only receives compiled dist output, reducing image size and build surface - Remove --insecure from default CMD; secure mode is now the default and --insecure must be passed explicitly as an opt-in - Re-add **/.git to .dockerignore to prevent .git dirs from being sent in the Docker build context (submodule init uses git submodule update inside the build stage, not the build context's .git)
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the container build/dev workflow into explicit multi-stage Docker targets (build, dev, prod) and updates Compose defaults to use the dev target and a slimmer dependency set, while adding ARM64/AMD64-aware Node/Go installs.
Changes:
- Replaces the single-stage-ish Docker build with
build/dev/prodstages and moves compilation (venv, UI build, agents) intobuild. - Adds
TARGETARCH-aware Node/Go downloads with checksum verification. - Updates
docker-compose.ymlto build thedevtarget by default and removes the deprecatedversion:key; adjusts.dockerignoreto keep.gitmodules.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| Dockerfile | Introduces multi-stage build (build/dev/prod), multi-arch toolchain installs, and copies built artifacts into runtime stages. |
| docker-compose.yml | Switches default build to target: dev, sets VARIANT: slim, and updates the run command. |
| .dockerignore | Stops excluding .gitmodules from the build context. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+70
to
+73
| # Ensure plugin submodules have been cloned | ||
| RUN git config --global --add safe.directory ${APP_DIR} \ | ||
| && git submodule sync --recursive \ | ||
| && git submodule update --init --recursive |
| @@ -8,7 +8,6 @@ Dockerfile | |||
| **/.git | |||
Comment on lines
+75
to
+79
| # Install Python dependencies, allowing failed installs for plugin requirements | ||
| RUN pip install --upgrade pip \ | ||
| && sed -i '/^lxml.*/d' ${APP_DIR}/requirements.txt \ | ||
| && pip install -r ${APP_DIR}/requirements.txt \ | ||
| && find ${APP_DIR}/plugins/ -type f -name 'requirements.txt' -print0 | xargs -0 -n1 pip install --no-cache-dir -r || true |
Comment on lines
+75
to
+80
| # Install Python dependencies, allowing failed installs for plugin requirements | ||
| RUN pip install --upgrade pip \ | ||
| && sed -i '/^lxml.*/d' ${APP_DIR}/requirements.txt \ | ||
| && pip install -r ${APP_DIR}/requirements.txt \ | ||
| && find ${APP_DIR}/plugins/ -type f -name 'requirements.txt' -print0 | xargs -0 -n1 pip install --no-cache-dir -r || true | ||
|
|
| ENV PATH="${VENV_DIR}/bin:${PATH}" | ||
| ENV PATH="/usr/local/lib/node/bin:${PATH}" | ||
|
|
||
| CMD ["python3", "/usr/src/app/server.py"] |
Contributor
There was a problem hiding this comment.
Pull request overview
Refactors the container build setup to support multi-stage Docker builds (build/dev/prod), improve runtime hardening, and enable multi-arch (amd64/arm64) builds while adjusting the default dev workflow to run via docker compose.
Changes:
- Replaced the single-stage runtime Dockerfile with named
build,dev, andprodstages and added direct Node/Go installs keyed byTARGETARCH. - Updated
docker-compose.ymlto target thedevstage and switched the default build variant toslim. - Updated
.dockerignoreto stop excluding.gitmodulesfrom the build context.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| Dockerfile | Introduces build/dev/prod stages, installs Node/Go by architecture, builds UI and agents in build stage, hardens prod runtime user. |
| docker-compose.yml | Targets dev stage, uses slim build arg, runs server with --build, removes deprecated version: key. |
| .dockerignore | Keeps .gitmodules in build context to support submodule-related workflows. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+118
to
+141
| #----( Dev Stage )-------------------------------- | ||
| FROM python:${PYTHON_VERSION}-slim-bookworm AS dev | ||
|
|
||
| # Set timezone (default to UTC) | ||
| ARG TZ="UTC" | ||
| RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \ | ||
| echo $TZ > /etc/timezone | ||
|
|
||
| # Install pip requirements | ||
| RUN pip3 install --break-system-packages --no-cache-dir -r requirements.txt | ||
| # Install caldera dependencies | ||
| RUN apt-get update -qy \ | ||
| && apt-get --no-install-recommends -y install git curl ca-certificates unzip mingw-w64 zlib1g \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| # For offline atomic (disable it by default in slim image) | ||
| # Disable atomic if this is not downloaded | ||
| RUN if [ ! -d "/usr/src/app/plugins/atomic/data/atomic-red-team" ] && [ "$VARIANT" = "full" ]; then \ | ||
| git clone --depth 1 https://github.com/redcanaryco/atomic-red-team.git \ | ||
| /usr/src/app/plugins/atomic/data/atomic-red-team; \ | ||
| else \ | ||
| sed -i '/\- atomic/d' conf/default.yml; \ | ||
| fi | ||
| COPY --from=build /usr/local/go /usr/local/go | ||
| COPY --from=build /usr/local/lib/node /usr/local/lib/node | ||
| COPY --from=build /usr/local/venv /usr/local/venv | ||
|
|
||
| # For offline emu | ||
| # (Emu is disabled by default, no need to disable it if slim variant is being built) | ||
| RUN if [ ! -d "/usr/src/app/plugins/emu/data/adversary-emulation-plans" ] && [ "$VARIANT" = "full" ]; then \ | ||
| git clone --depth 1 https://github.com/center-for-threat-informed-defense/adversary_emulation_library \ | ||
| /usr/src/app/plugins/emu/data/adversary-emulation-plans; \ | ||
| fi | ||
| ENV APP_DIR=/usr/src/app | ||
| ENV VENV_DIR=/usr/local/venv | ||
| ENV PATH="/usr/local/go/bin:${PATH}" | ||
| ENV PATH="${VENV_DIR}/bin:${PATH}" | ||
| ENV PATH="/usr/local/lib/node/bin:${PATH}" | ||
|
|
||
| # Download emu payloads | ||
| # emu doesn't seem capable of running this itself - always download | ||
| RUN cd /usr/src/app/plugins/emu; ./download_payloads.sh | ||
| WORKDIR ${APP_DIR} |
| ENV PATH="${VENV_DIR}/bin:${PATH}" | ||
| ENV PATH="/usr/local/lib/node/bin:${PATH}" | ||
|
|
||
| CMD ["python3", "/usr/src/app/server.py"] |
| esac; \ | ||
| curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${node_arch}.tar.xz" -o /tmp/node.tar.xz; \ | ||
| curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/SHASUMS256.txt" -o /tmp/node.SHASUMS256.txt; \ | ||
| grep "node-v${NODE_VERSION}-linux-${node_arch}.tar.xz" /tmp/node.SHASUMS256.txt | sha256sum -c -; \ |
Comment on lines
+70
to
+73
| # Ensure plugin submodules have been cloned | ||
| RUN git config --global --add safe.directory ${APP_DIR} \ | ||
| && git submodule sync --recursive \ | ||
| && git submodule update --init --recursive |
Comment on lines
+75
to
+77
| # Install Python dependencies, allowing failed installs for plugin requirements | ||
| RUN pip install --upgrade pip \ | ||
| && sed -i '/^lxml.*/d' ${APP_DIR}/requirements.txt \ |
Comment on lines
+94
to
+97
| # Fetch atomic data or disable it in slim | ||
| RUN if [ "$VARIANT" = "full" ] && [ ! -d "${APP_DIR}/plugins/atomic/data/atomic-red-team" ]; then \ | ||
| git clone --depth 1 https://github.com/redcanaryco/atomic-red-team.git ${APP_DIR}/plugins/atomic/data/atomic-red-team; \ | ||
| else \ |
5 tasks
The else branch previously ran for both VARIANT=slim and when VARIANT=full with an existing atomic data dir, unintentionally removing atomic from conf/default.yml in full builds.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR modernizes the container workflow by converting the single Dockerfile into explicit build/dev/prod stages, updating docker-compose.yml to use the dev target by default, and adjusting .dockerignore to better support submodule builds and multi-arch images.
Changes:
- Replaces the prior Docker build flow with multi-stage
build,dev, andprodtargets and adds arch-aware Node/Go installation. - Updates docker-compose to target the
devstage and switches default build variant toslim. - Updates
.dockerignoreto stop excluding.gitmodules.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| Dockerfile | Introduces multi-stage build/development/production targets, multi-arch toolchain installs, venv reuse, and a non-root prod runtime. |
| docker-compose.yml | Targets dev stage, removes deprecated version, sets default VARIANT to slim, updates command. |
| .dockerignore | Stops ignoring .gitmodules to support submodule-related build steps. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+70
to
+73
| # Ensure plugin submodules have been cloned | ||
| RUN git config --global --add safe.directory ${APP_DIR} \ | ||
| && git submodule sync --recursive \ | ||
| && git submodule update --init --recursive |
| ENV PATH="${VENV_DIR}/bin:$PATH" | ||
| ENV PATH="/usr/local/lib/node/bin:${PATH}" | ||
|
|
||
| ADD . ${APP_DIR} |
Comment on lines
+75
to
+79
| # Install Python dependencies, allowing failed installs for plugin requirements | ||
| RUN pip install --upgrade pip \ | ||
| && sed -i '/^lxml.*/d' ${APP_DIR}/requirements.txt \ | ||
| && pip install -r ${APP_DIR}/requirements.txt \ | ||
| && find ${APP_DIR}/plugins/ -type f -name 'requirements.txt' -print0 | xargs -0 -n1 pip install --no-cache-dir -r || true |
| ENV PATH="${VENV_DIR}/bin:${PATH}" | ||
| ENV PATH="/usr/local/lib/node/bin:${PATH}" | ||
|
|
||
| CMD ["python3", "/usr/src/app/server.py"] |
| ENV PATH="${VENV_DIR}/bin:${PATH}" | ||
| ENV PATH="/usr/local/lib/node/bin:${PATH}" | ||
|
|
||
| CMD ["python3", "/usr/src/app/server.py"] |
| volumes: | ||
| - ./:/usr/src/app | ||
| command: --log DEBUG | ||
| command: ["python", "/usr/src/app/server.py", "--build"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
build(compiles everything),dev(mounts source via volume for local development), andprod(hardened runtime with a non-rootappuser)TARGETARCH-aware selectors foramd64/arm64buildstage and copied intodev/prodviaCOPY --from=build, keeping the runtime image leanupdate-agents.shif Go is availabledocker-compose.ymlupdated: removes deprecatedversion:key, targets thedevstage, usesslimvariant by default (full offline deps are only needed inprod).dockerignorecorrected to stop excluding.gitmodules(which is required for submodule initialization during the build)CMDuses--insecureflag in prod stage; dev compose command uses--buildNotes / Issues to verify before merge
ARG GO_VERSION=1.25.4— Go 1.25.4 did not exist as of the branch date (Nov 2025); latest stable at that time was 1.23.x. This should be verified and corrected before merge to avoid a broken builddevstage has no CMD/ENTRYPOINT; it relies entirely ondocker-compose.ymlproviding the run command — this is intentional for the dev workflow but should be documentedemu'sdownload_payloads.shis no longer called during build; previously it was always run in the container — confirm whether this is intentional for the new flow--insecure; review whether this is appropriate for a production imageTest plan
1.25.4exists or updateARG GO_VERSIONto a valid releaseprodtarget:docker build --target prod .devtarget:docker build --target dev .docker compose upand confirm Caldera starts correctlydocker buildx build --platform linux/arm64 --target prod ..gitmodulesno longer excluded from.dockerignore)